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(57) Abstract 

In ATM networks, digital data in ATM cells arc sent to a destination node over more than one transmission link in round robin 
fashion. This is called inverse multiplexing. At connection start-up, the source node informs the destination node of the specific round 
robin fashion of the transmission links so that the ATM cells are reassembled in a proper sequential order. 
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ATM INVERSE MULTIPLEXING 



Tpchnical Field and Indus trial Application 

The invention relates to a new mechanism for sending ATM 
5 cells transparently over multiple slower transmission links. In 
particular, the invention is directed to a method of inverse 
multiplexing of a series of ATM cells transparently over N 
transmission links (N being a positive integer) of slower speed. 

10 Background Art 

It has been recognized that the Tl/El rate (1.544/2.048 Mbit/s) is a 
cost effective way of user access to an ATM network as well as 
connection between ATM network switches. However, as ATM 
technology for wide area networks is deployed more and more, 
15 demands for transmission links of a rate higher than Tl/El are 

increasing. Links of higher rates, such as T3/E3 (44.736/34.368 Mbit/s), 
have been designed to meet these needs. However, the cost of T3/E3 
links is still prohibitive in many cases and the ratio of cost versus 
realistic utilization of the entire rate is not always attractive and fully 
20 justified for new ATM end users and service providers. ATM inverse 
multiplexers (ATMs) have been proposed to satisfy the need by using 
multiple Tl/El links which are grouped collectively to provide the 
service at a higher rate. 

Figure 1 and Figure 2 show two sample configurations in which 
25 AIMs are used. Figure 1 depicts a user access to a network through 
user network interfaces (UNIs) and Figure 2 a link connection between 
ATM switches through broadband inter-carrier interfaces (BICIs) or 
private network to network interfaces (PNNIs). 

Referring to the figures, the basic function of AIMs is to work in 
30 pairs to take an ATM cell stream coming from the ATM layer, send it 
over the multiple links by spreading cells over the available links and 
ensure that the initial cell stream can be retrieved at the far end. Thus 
the AIMs preferably make the ATM traffic transparent to the ATM 
layer over multiple links which connect them. As far as the ATM layer 
35 is concerned, it should only see a pipe whose rate is now the sum of the 
multiple link rates. It is assumed that each link is run in clear-mode 
without the presence of intermediate ATM nodes processing ATM 



WO 96/17489 




PCT/CA95/00322 



2 



cells. This means that there should be no cell discard by any 
intermediate transmission equipment. 

Currently no ATM inverse multiplexing protocols have been 
proposed which can properly interwork existing ATM inverse 
5 multiplexers or other ATM products which are already available on the 
market, and yet are flexible enough to fit into the current standard 
ATM specifications. Two proposals for an ATM inverse multiplexing 
protocol have so far been made and are described in detail below. 

10 New Transmission Convergence Protocol Using GFC Bits 

This protocol was presented in "Physical Layer Sub-Working 
Group ATM Forum/94-0775, ATM Inverse Multiplexing Mechanism", 
September 1994, by StrataCom Inc. The protocol robs two of the 
Generic Flow Control (GFC) bits contained in each cell transmitted 

15 over the multiple Tl/El links to implement a new transmission 

convergence (TC) layer. Figure 3 shows the ATM cell structure which 
is defined in the ITU Recommendation. The TC layer is defined by one 
GFC bit for framing and the other one for link control. The framing bit 
is used to determine relative link delays while the link control bit is 

20 used for communication, control and administration between two TC 
points at two ends of the inverse multiplexer. 

In order to establish the sequence of cells over the links in a 
round robin manner, one end is defined as being "master 1 ' and the 
other as "slave". The "master" decides and informs the slave about the 

25 multiple link configuration using the control channel implemented 
through the link control bits. 

This protocol is only applicable, however, for UNI application 
points because GFC bits that are robbed to implement the TC layer are 
only present in a cell defined for UNI. For NNI cells, the 

30 corresponding bits are no longer available since they are captured 

under the VPI field. Service providers are interested in ATM inverse 
multiplexers for carrying ATM traffic at rates higher than Tl/El and 
lower than T3/E3, but this protocol will not satisfy their need. It 
should also be noted that the protocol calls for a need to identify a 

35 "master" and a "slave" TC point and that requires an additional setting 
to be performed by the network operator. 
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Bit Pipe Inverse Multiplexing 

This protocol was presented in "Physical Layer Sub-Working 
Group ATM Forum/ 94-0956, Inverse Multiplexing of ATM cells over 
low speed UNIs such as Tl and El", September 1994, by Digital Link 
5 Corporation. It proposes a "bit pipe" inverse multiplexing technique 
requiring the definition of a "bonding" (bandwidth on demand) like 
specification for N (positive number) Tl/El inverse multiplexing. 

It is not clear in the proposal how both ends of the links 
exchange information concerning the order of cells to be transferred 
10 from one end to another over multiple links. The proposal mentions 
the existence and deployment of physical layer protocols that perform 
inverse multiplexing. The inverse multiplexer which can be used in 
this proposal is presumably the one defined by Digital Link 
Corporation in their "DL3800 DS1 Inverse Multiplexer Users Manual, 

15 1993". 

The inverse multiplexing protocol defined in the above user's 
manual relies on the definition of an extra bit taken from Tl/El 
payload bits to configure the multiple links and adjust differential Unk 
delays. This protocol introduces the need for extra processing of data 

20 between devices dealing with Tl/El frames and ATM cell delineation. 
It also causes the ATM cells to no longer be byte aligned with the 
DS1/E1 frame. This is a requirement by the ATM Forum UNI DS1/E1 
Physical Layer specifications. Changes like this would not be welcome 
by end users, vendors and service providers who are already using and 

25 deploying ATM equipment. 



Ohj ects of the Invention 

It is therefore an object of the invention to provide a method of 
sending ATM traffic over a connection consisting of a plurality of 

30 transmission links. 

It is another object of the invention to provide a method of 
sending a series of ATM cells spread over a plurality of transmission 
links in a specific round robin order. 

It is yet another object of the invention to provide a method of 
35 sending ATM cells containing sequence numbers therein over a 
plurality of transmission links in a specific round robin order. 
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It is a further object of the invention to provide a method of 
sending a series of ATM cells between ATM inverse multiplexers over 
a connection consisting of a plurality of transmission links, transparent 
to the ATM layer. 

5 It is yet another object of the invention to provide a method of 

sending ATM traffic over a connection which consists of a plurality of 
transmission links and has been reconfigured after the connection 
start-up. 

It is still another object of the invention to provide an ATM 
10 inverse multiplexing method which is applicable to UNIs, BICIs and 
PNNs. 

It is a further object of the invention to provide a method of 
preserving link integrity by periodically sending sequence number 
cells. 

It is a further object of the invention to provide a method of 
handling link failure and link reconfiguration. 

Disclosure of the Invention 

Briefly stated, according to one aspect the invention is directed to 
a method of sending digital data from a source node to a destination 
node over a connection consisting of a plurality of transmission links, 
the digital data containing a series of ATM data cells. The method 
comprises steps of the source node informing the destination node, at 
the connection start-up by way of ATM SN cells, of a specific round 
robin order in which the series of ATM data cells are to be transmitted 
over the connection, and sending each ATM data cell in the series to 
the destination node in the round robin order. 

According to a further aspect of the invention, the method 
comprises a further step of the source node sending ATM SN cells 
whose sequence number field is set to indicate the specific round robin 
order. 

According to yet another aspect, the invention comprises 
repeating the steps which have been performed at the connection start- 
up, wherever the connection is to be reconfigured. 

According to still another aspect, the invention comprises a step 
of a node informing another node of a link failure and repeating the 
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steps which have been performed at the connection start-up, wherever 
the connection is to be reconfigured. 

Brief Description of the Drawings 
5 For a more complete understanding of the invention and for 

further objects and advantages thereof, reference may now be made to 

the following description, taken in conjunction with the accompanying 

drawings, in which: 

Figure 1 shows a sample configuration involving AIM UNIs; 
10 Figure 2 shows a sample configuration involving AIM BICIs or 

PNNIs; 

Figure 3 depicts the ATM cell structure defined in the ITU 
Recommendation; 

Figure 4 is a schematic illustration of multiplexing and de- 
15 multiplexing of ATM cells over AIMs and links; 

Figure 5 shows preassigned cell header values for use by the 
physical layer; 

Figure 6 shows a header pattern of an AIM OAM cell, according 
to one embodiment of the invention; 
20 Figure 7 indicates allocation of OAM functions in the 

information field according to one embodiment of the invention; 

Figure 8 is a schematic illustration of multiplexing and 
demultiplexing of AIM OAM cells according to one embodiment of the 
invention; and 

25 Figure 9 is a schematic illustration of periodic multiplexing and 

demultiplexing of AIM OAM cells during transmission of ATM data 
cells according to one embodiment of the invention. 

Mode(s) of Carrying Out the Invention 
30 There are requirements that have to be considered when 

defining a new ATM inverse multiplexing protocol. These 
requirements are: 

• It must multiplex and demultiplex an ATM cell stream distributed 
in a round robin manner over multiple links such as Tl/El links. 
35 • It must adjust up to 32 milliseconds of differential link delays 
between individual links in a case where Tl/El links are used. 
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• It must reconfigure multiple links in the event that a link has to be 
added, deleted or is considered inadequate to provide service. 

• It must be defined for not only UNIs but also for PNNIs and BICIs. 

• It must be transparent to the devices handling the convergence of 
5 ATM cells into the PDH signal. 

• It must be transparent to the devices dealing with ATM layer cells. 

The present invention achieves all of the above requirements 
and solves the problems discussed earlier. The invention relates to a 
new ATM inverse multiplexing scheme that makes use of a physical 

10 layer operation administration and maintenance (OAM) cell which has 
been properly defined. This OAM cell is defined to contain valuable 
information to allow proper operation of the ATM inverse 
multiplexing mechanism and also to provide opportunity for handling 
a link failure situation. 

15 The newly defined OAM cell is called an AIM OAM cell or AIM 

Sequence Number (SN) cell and is mainly designed to carry a cell 
sequence number and a feedback link status field. The sequence 
number in the SN cell is made available for the receiver end for 
recovering the initial cell stream from the incoming links. The 

20 feedback link status is made available for the receiver to inform the 
transmitter, by sending its own SN cells, that it is receiving cells and it 
is also an integral part of the same round robin mechanism, that is to 
say, the feedback link status value SN cells in either direction must 
agree with each other, although the sequence numbers at both ends 

25 may be different. When the receiver sends its own SN cells in 
response to the transmitter that it is receiving cells, it is in fact 
acknowledging that the receiver is ready to receive subsequent ATM 
data cells. 

Figure 4 shows how the ATM cells are multiplexed and then 
30 demultiplexed over AIMs in one direction. At the transmitting node, 
an AIM 10 takes a series of ATM cells from an ATM layer device. It 
spreads ATM cells and transmits each cell over each of N transmission 
links, N being a positive integer. The order of transmission is in round 
robin fashion. This process is called an inverse multiplexing. At the 
35 receiving node, cells from N links are inverse demultiplexed 

(assembled) and sent to an ATM layer device by an AIM 14. The same 
order must be employed at this node to recover a proper sequence of 
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cells. Both nodes must be aware of the round robin order which is to 
be employed. Upon initialization, therefore, both AIMs send a series of 
SN cells in round robin fashion over the links, e.g. Tl/El links. This 
allows the receiver AIM at both nodes to establish the sequence in 
5 which to read cells from the incoming links as well as to adjust relative 
link delay. 

As mentioned above, the SN cell also carries an extra field used 
by each Tl/El link to indicate that both AIMs belong to the same link 
round robin. This information can then be used locally to determine if 

10 at the corresponding local node a link should be added, removed or 
maintained in the round robin. 

When a change of link configuration occurs by a link being 
added, removed or declared as being down, each node sends a series of 
SN cells to allow the far-end node to reestablish the sequence of cells to 

15 read from the incoming links. 

The protocol of the invention calls for a physical layer OAM cell 
that is defined to be exclusively processed by the ATM inverse 
multiplexers. The new cell structure has to be consistent with the cell 
structure defined in ITU Recommendation 1.361. Figure 3 shows that 

20 structure of UNI/NNI ATM cells that is defined in 1.361. ITU 

Recommendations 1.361 and 1.432 state that ATM cells consisting of a 
header whose VPI and VCI fields are set to zero are reserved for use by 
the physical layer. So far three preassigned values of the cell header are 
reserved for use by the physical layer. They are shown in Figure 5. 

25 The physical layer OAM cell according to one embodiment of the 

invention is then defined by using a non-assigned value by setting the 
FT field to "111". This is shown in Figure 6. 

The AIM OAM cell payload is then available for exchanging 
information between AIMs. The cell payload consists of a series of 

30 fields whose locations are shown in Figure 7. 

The following fields are identified for the AIM protocol 
according to one embodiment of the invention: 

• ATM inverse multiplexing remote defect indicator (AIMRDI) - one 
octet is allocated and the proposed coding is all "1". 
35 • ATM inverse multiplexing far-end receiver ready (AIMFERR) - one 
octet is allocated and the proposed coding is all "1". 
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♦ ATM inverse multiplexing cell sequence number (AIMCSN) - it is 
defined to contain the sequence number of the cells sent over the 
multiple links handled by the ATM inverse multiplexers. It is 
designed so as to have a sufficiently large cycle to allow the ATM 

5 inverse multiplexer to absorb link delays of up to 32 milliseconds. 
There are 16 bits allocated to the AIMCSN field. The counting is 
then done modulo 65536. 

• Cell error control (CEC) - is used to detect errors in the cell payload. 
It is proposed to CRC-10 as proposed in ITU Recommendation 1.432. 

10 • Reserve field (R) - contains the octet pattern of "01101010", which is 
the same as that of the idle cell as proposed in ITU Recommendation 
1.361. 

Referring to Figure 8, the ATM inverse multiplexing protocol 
according to one embodiment is described in detail below. The figure 

15 only shows one direction for clarity. It is shown that transmission 

links are all Tl/El links but, of course, links can have a different speed 
from Tl/El, as long as all the links have the same speed. A 
transmitting node 20 collects digital data consisting of a series of ATM 
data cells from the ATM layer. An ATM inverse multiplexer (AIM) at 

20 the transmitting node spreads over multiple links 24 for transmission 
to a receiving node 26. An AIM at the receiving node reassembles 
ATM data cells received from the links in proper order and sends them 
to the ATM layer. 

25 Link Start-up 

Upon connection start-up, AIMs at both nodes start inserting 
AIM OAM cells (AIM SN cells) carrying cell sequence number over the 
available links in round robin fashion. The sequencing of cells is based 
on the order in which cells have to be transmitted on the virtual link 

30 (composed of N physical links). However, the sequence number is 
only carried over the SN cells. This sequence number assignment 
allows the receiving AIM to retrieve the original cell sequence. The 
receiving node queues the received AIM SN cells until it determines 
the sequence in which to read the ATM data cells from the incoming 

35 links as well as the differential delay among individual links. Then, it 
starts sending AIM SN cells of its own, with the AIMFERR field set to 
"one" for each link which is now considered "ready" to receive ATM 
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traffic. From that moment, the receiving node knows the sequence of 
cells coming from the links. A link is being considered available if cells 
are currently delineated and AIM-RDI is not received on the incoming 
link. 

5 When the transmitting node starts sending the AIM SN cells, it 

starts a time-out of 100 milliseconds within which the receiving node 
determines the sequence of AIM SN cells which it is receiving from the 
links. The time-out expires unless the transmitting node receives the 
"ready" signal from all the links which were considered available at the 

10 initialization. 

If there is no "ready" link when the time-out expires, the local 
node reevaluates the availability of the links (using cell delineation), 
starts sending AIM SN cells over the available links and re-starts the 
time-out. This procedure is repeated until at least one available link is 

15 declared ready. 

When the time-out has been canceled due to the reception of the 
"ready" signal from all the available links, or when there is at least one 
link ready when the time-out expires, the local end starts sending ATM 
layer cells over the "ready" link or links using the same round robin 
20 order used at the time of initialization. 

Each ATM layer cell gets a sequence number assigned to it, but 
only the AIM SN cells carry that number across the links. After both 
nodes have started sending ATM layer cells, they periodically send a 
series of "n" AIM SN cells over links to allow the receiving node to 
25 readjust the differential delays among the links. The value of "n" is 
equal to the number of "ready" links used to carry ATM layer cells. 
This means that the receiving node always scans for AIM SN cells in 
order to determine if it is always reading cells from the multiple links 
in the correct order. How often a burst of "n" AIM SN cells is sent 
30 depends upon the link utilization but the maximum period has been 
set to 50 milliseconds in one embodiment. An example of cells sent 
over a virtual link composed of a plurality of transmission links is 
shown in Figure 9. 



35 Link Reconfiguration 

As mentioned above, the protocol according to the invention 
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also deals with link reconfiguration. There are three possible cases in 
which reconfiguration can occur: 

• a new link has to be added to the round robin; 

• a link has to be intentionally removed from the round robin; and 
5 • a link has been declared inadequate to provide service (e.g. link 

down by failure etc.). 

In the first two cases, a node starts link reconfiguration by 
performing the same initialization process for a connection start-up, 
that is to say, it chooses a round robin order among the links and starts 

10 sending AIM SN cells using the chosen round robin order. The 
receiving node, seeing the occurrence of AIM SN cells, will stop 
transmitting traffic and starts sending AIM SN cells of its own while 
adjusting itself to receive traffic from the incoming links. The rest of 
the protocol is as described earlier for start-up. 

15 Two possible cases for reporting failure conditions are: 

1) A medium problem reported through Loss of Delineation 
(LCD) failure condition. In this case, the corresponding link should not 
be used for service. When detecting LCD, SN cells with AIMRDI set to 
"1" should be sent over the corresponding outgoing link. SN cells 

20 containing AIM-RDI do not carry a valid sequence number. At the far- 
end, the detection of AIM-RDI signals will indicate that the link is not 
to be used. Therefore, if LCD or AIM-RDI is detected, the link is no 
longer considered available and won't be allowed to be part of the 
round robin on both sides. When this has happens, SN cells are re- 

25 sent over the remaining links to re-establish the connection between 
the two ends. 

2) Ceils are lost without an LCD or AIM-RDI being reported. For 
instance, this would occur when a few cells are discarded by the 
physcial layer device due to bit errors in cell headers. This would cause 

30 the cell sequencing to be affected. One symptom would be the detection 
of SN cells whose number is no longer the same as that expected (since 
one or more of the previous cells are missing). In this case, the 
problem might be partially corrected by re-ajusting the receiver 
buffering system (assuming the difference between the SN cell number 

35 and the expected number is small). The other symptom would be the 
absence of a SN cell on one link when getting a burst of SN cells on all 
the other links. In that case, the local end would have to force the links 
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to be re-configured. If the symptoms described above reoccur over a 
given period, the bad link may have to be removed from the round 
robin. 

A link reconfiguration also occurs when it takes too much time 
5 for one link to receive cells from the far-end (receiving) node, that is to 
say, no cells within, e.g., 32 milliseconds. 

Cell Sequence Numbe r Range 

Because an ATM inverse multiplexer must absorb a differential 
10 delay between individual links of up to 32 milliseconds in one 

embodiment, and because the system needs to deal with a maximum of 
8 Tl/El links, it is necessary to have a sequence number whose modulo 
is large enough to accommodate such delay. 

As a practical example, the following parameters for El are 

15 considered: 

• full rates: 2.048 Mbit/ s 

• payload rates: 30/32*2.048 Mbit/s = 1.92 Mbps 

• ATM cell: 53 bytes 

• ATM cell period time: 221 microseconds 
20 • ATM cells/32 milliseconds = 144 cells. 

For a delay of up to 32 milliseconds on each link, there is a need 
to queue cells for at least the same length of time on each link. 
Therefore, in this example, this means that a delay of up to a period of 
144 cells between two links is possible. An ATM inverse multiplexer 

25 can handle a maximum of 8 Tl/El links. Therefore up to 1008 cells 
(144*7 cells) must be queued at one time by one ATM inverse 
multiplexer. This requires a sequence number modulo large enough to 
cover this scenario. A simple case in one embodiment is to use a 16-bit 
count that has modulo 65536. 

30 The ATM inverse multiplexing protocol according to the 

invention realizes the following characteristics: 

• it is applicable to UNIs, BICIs and PNNIs (any applicable points in 
an ATM network); 

• it does not affect the ATM cell header of currently defined cells; 
35 • it does not require a change to current ATM physical layer devices 

dealing with the convergence of ATM cells into Tl/El; 

• it operates transparently to the ATM layer; 
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• it does not require a change to current devices dealing with the 
processing of the ATM layer cell; 

• it is self-configuring among available links upon start-up and self- 
reconfiguring among the multiple links in the case when a new 
link has to be added, deleted or considered inadequate to provide 
service; and 

• it includes a sequence number whose modulo is large enough to 
meet the requirement of a large differential delay among the links. 
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WHAT IS CLAIMED IS: 

1. In a method of sending digital data from a source node to a 
destination node over a connection consisting of a plurality of 

5 transmission links, said data containing a series of ATM data cells, the 

invention being characterized in steps of: 

at the connection start-up, the source node sending to the 

destination node ATM SN cells mforming the latter of a specific order 

in which the series of ATM data cells are to be transmitted over the 
10 connection; 

the destination node sending ATM SN cells whose receive ready 

field is set to indicate that the destination node is ready to receive ATM 

data cells in specific round robin order from the plurality of 

transmission links; and 
15 the source node sending each ATM data cell in said series to the 

destination node in said round robin order. 

2. The method according to claim 1 wherein the step of the 
source node informing the destination node includes a step of the 

20 source node sending a series of ATM SN ceUs whose sequence number 
field is set in a sequential order, each ATM SN cell being sent over each 
transmission link in a specific round robin order. 

3. The method according to claim 2 wherein the step of the 
25 destination node sending the source node includes a step of the 

destination node sending a series of ATM SN cells whose receive ready 
field is set and whose sequence number field is set in a sequential 
order, each ATM SN cell being sent over each transmission link in a 
specific round robin order. 

30 

4. The method according to claim 3 comprising a further step of 
the source node and destination node sending periodically to one 
another a series of ATM SN cells over the transmission links in 
specific round robin order within a predefined time period. 
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5. The method according to claim 4 wherein said ATM SN cells 
are physical layer OAM cells whose protocol fields are properly set to 
indicate types of cells and said sequential order. 

5 6. The method according to claim 5 wherein said ATM SN cells 

are physical layer OAM cells whose protocol fields contain a field to 
indicate the destination node is ready to receive ATM data cells. 

7. The method according to claim 4 wherein the source node 
10 and destination node send periodically a burst of "n" SN cells to each 

other over "n" transmission links, within a predetermined maximum 
time period. 

8. The method according to claim 7 wherein the period of the 
15 source node and destination node sending a burst of "n" SN cells to 

each other over "n" transmission links is adjustable within the 
maximum time period, depending upon the link utilization. 

9. In a method of sending digital data from a node to another 
20 node over a connection consisting of a plurality of transmission links, 

said data containing a series of ATM data cells, the invention being 
characterized in steps of: 

whenever the connection is to be reconfigured, a first node 
sending to the other node ATM SN cells whose sequence number field 
25 is set to indicate a specific round robin order in which the series of 
ATM data cells are to be transmitted over the connection; 

the other node, in response to the first node, sending ATM SN 
cells whose receive ready field is set to indicate the other node is ready 
to receive ATM data cells in specific round robin order from the 
30 plurality of transmission links; and 

the first node sending each ATM data cell in said series to the 
other node in said round robin order. 

10. A method according to claim 9 comprising further steps of: 
35 whenever the connection is to be reconfigured upon failure of a 

transmission link, a node sending to the other node an indication that 
a link has failed; and 
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the two nodes performing the steps recited in claim 9 over the 
remaining transmission links. 

11. The method according to claim 9 wherein said ATM SN cells 
5 are physical layer OAM cells whose protocol fields are properly set to 
indicate types of cells and said sequential order. 
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12. The method according to claim 10 wherein said ATM SN 
cells are physical layer OAM cells whose protocol fields are properly set 
to indicate the link failure. 
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